<html>
<head><meta charset="utf-8"><title>eager expansion · t-compiler · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/index.html">t-compiler</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/eager.20expansion.html">eager expansion</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="157679182"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/eager%20expansion/near/157679182" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alexander Regueiro <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/eager.20expansion.html#157679182">(Feb 06 2019 at 10:16)</a>:</h4>
<p>This is to discuss macro eager expansion, in particular w.r.t. <span class="user-mention" data-user-id="201856">@Edward Pierzchalski</span>'s recent RFC and @nrc's older one.</p>



<a name="157679195"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/eager%20expansion/near/157679195" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alexander Regueiro <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/eager.20expansion.html#157679195">(Feb 06 2019 at 10:17)</a>:</h4>
<p>Ah... I don't think @nrc is on Zulip</p>



<a name="157679889"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/eager%20expansion/near/157679889" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edward Pierzchalski <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/eager.20expansion.html#157679889">(Feb 06 2019 at 10:31)</a>:</h4>
<p>For future reference, here are links to the <a href="https://github.com/rust-lang/rfcs/pull/2320" target="_blank" title="https://github.com/rust-lang/rfcs/pull/2320">eRFC</a> by me and the <a href="https://github.com/nrc/rfcs/blob/eager-expand/text/0000-eager.md" target="_blank" title="https://github.com/nrc/rfcs/blob/eager-expand/text/0000-eager.md">RFC</a> by nrc.</p>



<a name="157712999"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/eager%20expansion/near/157712999" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alexander Regueiro <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/eager.20expansion.html#157712999">(Feb 06 2019 at 18:29)</a>:</h4>
<p>Yep, good.</p>



<a name="157713110"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/eager%20expansion/near/157713110" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alexander Regueiro <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/eager.20expansion.html#157713110">(Feb 06 2019 at 18:30)</a>:</h4>
<p>So... apparently @nrc is leaving Rust or at least hugely decreasing participation.</p>



<a name="157713126"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/eager%20expansion/near/157713126" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alexander Regueiro <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/eager.20expansion.html#157713126">(Feb 06 2019 at 18:30)</a>:</h4>
<p><span class="user-mention" data-user-id="201856">@Edward Pierzchalski</span></p>



<a name="157713183"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/eager%20expansion/near/157713183" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alexander Regueiro <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/eager.20expansion.html#157713183">(Feb 06 2019 at 18:31)</a>:</h4>
<p>We could have a chat with <span class="user-mention" data-user-id="123856">@Vadim Petrochenkov</span> about this though, who is sort of the macros guru.</p>



<a name="157713216"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/eager%20expansion/near/157713216" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alexander Regueiro <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/eager.20expansion.html#157713216">(Feb 06 2019 at 18:31)</a>:</h4>
<p>Are you free for a meeting some time next week? :-)</p>



<a name="157726503"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/eager%20expansion/near/157726503" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edward Pierzchalski <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/eager.20expansion.html#157726503">(Feb 06 2019 at 21:11)</a>:</h4>
<p>I should be pretty free next week for a meeting. However, if I recall correctly petrochenkov and jseyfried were too busy or otherwise not keen on participate much on this. Should we pull in <span class="user-mention" data-user-id="126931">@centril</span> or llogic?</p>



<a name="160526593"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/eager%20expansion/near/160526593" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edward Pierzchalski <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/eager.20expansion.html#160526593">(Mar 11 2019 at 23:10)</a>:</h4>
<p>For anyone interested, after discussion between <span class="user-mention" data-user-id="124069">@Alexander Regueiro</span> and myself, the RFC has been polished and updated (and might enter FCP soonish?), so if you have any feedback now's the time!</p>



<a name="177145790"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/eager%20expansion/near/177145790" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/eager.20expansion.html#177145790">(Oct 02 2019 at 12:47)</a>:</h4>
<p>Hi <span class="user-mention" data-user-id="201856">@Edward Pierzchalski</span> , I have some questions to pose while I write my suggested edits to the RFC</p>



<a name="177145923"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/eager%20expansion/near/177145923" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/eager.20expansion.html#177145923">(Oct 02 2019 at 12:49)</a>:</h4>
<p>(but I wanted to double-check that you are active here before writing them out in this topic)</p>



<a name="177150136"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/eager%20expansion/near/177150136" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edward Pierzchalski <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/eager.20expansion.html#177150136">(Oct 02 2019 at 13:35)</a>:</h4>
<p>Sure, go ahead (but I'm just about to go to bed, so I'll respond later)</p>



<a name="177150236"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/eager%20expansion/near/177150236" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/eager.20expansion.html#177150236">(Oct 02 2019 at 13:36)</a>:</h4>
<p>okay, first question (and any of these may reflect gross ignorance on my part)</p>



<a name="177150291"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/eager%20expansion/near/177150291" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/eager.20expansion.html#177150291">(Oct 02 2019 at 13:36)</a>:</h4>
<p><span aria-label="question" class="emoji emoji-2753" role="img" title="question">:question:</span>  Do you expect anyone using the API to not immediately call <code>.await</code> after invoking <code>eb.expand()</code>?</p>



<a name="177150348"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/eager%20expansion/near/177150348" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/eager.20expansion.html#177150348">(Oct 02 2019 at 13:37)</a>:</h4>
<p>I'm basically trying to imagine what my code would look like, and I don't understand what the value is of exposing the <code>Future</code> by returning it, versus having it call <code>.await</code> itself within the implementation of <code>fn expand(self)</code></p>



<a name="177150640"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/eager%20expansion/near/177150640" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/eager.20expansion.html#177150640">(Oct 02 2019 at 13:40)</a>:</h4>
<p><span aria-label="question" class="emoji emoji-2753" role="img" title="question">:question:</span> <code>ExpansionBuilder::from_tokens(tokens: TokenStream) -&gt; Result&lt;Self, ParseError&gt;</code> takes <code>tokens</code> by value, and tries to parse it. What is it going to parse it as: An expression? Or just as a token-tree?</p>



<a name="177150654"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/eager%20expansion/near/177150654" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/eager.20expansion.html#177150654">(Oct 02 2019 at 13:40)</a>:</h4>
<p>(I'm guessing "token-tree", but I wanted to double-check.)</p>



<a name="177150831"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/eager%20expansion/near/177150831" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/eager.20expansion.html#177150831">(Oct 02 2019 at 13:42)</a>:</h4>
<p>(or maybe its expecting a forest of token trees? namely, I don't yet know how it handles trailing input after the first token tree it parses)</p>



<a name="177150924"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/eager%20expansion/near/177150924" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/eager.20expansion.html#177150924">(Oct 02 2019 at 13:43)</a>:</h4>
<p>One goal I have is to try to revise the text so that we can illustrate the API by showing the <em>definition</em> of the declarative <code>expand!</code> in an appendix.</p>



<a name="177151124"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/eager%20expansion/near/177151124" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/eager.20expansion.html#177151124">(Oct 02 2019 at 13:45)</a>:</h4>
<p>That is, instead of saying "here's some useful functionality that is specified by this RFC" (and leaving it unclear whether this is some fundamental new feature being supplied), I'm anticipating it could say "the <code>ExpansionBuilder</code> API is all you need. For example, check out the definition of the declarative <code>expand!</code>,  given in Appendix Z."</p>



<a name="177202292"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/eager%20expansion/near/177202292" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edward Pierzchalski <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/eager.20expansion.html#177202292">(Oct 02 2019 at 23:27)</a>:</h4>
<blockquote>
<p><span aria-label="question" class="emoji emoji-2753" role="img" title="question">:question:</span>  Do you expect anyone using the API to not immediately call <code>.await</code> after invoking <code>eb.expand()</code>?</p>
</blockquote>
<p>That's a fair point. I suppose the other way of making it clear to the caller that <code>expand</code> is a potentially blocking, long-running operation is to... put it in the docs? One situation where that might backfire is when your proc macro does IO - then you want to be very clear about where such long-running things occur.</p>



<a name="177203257"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/eager%20expansion/near/177203257" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edward Pierzchalski <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/eager.20expansion.html#177203257">(Oct 02 2019 at 23:44)</a>:</h4>
<blockquote>
<p><span aria-label="question" class="emoji emoji-2753" role="img" title="question">:question:</span> <code>ExpansionBuilder::from_tokens(tokens: TokenStream) -&gt; Result&lt;Self, ParseError&gt;</code> takes <code>tokens</code> by value, and tries to parse it. What is it going to parse it as: An expression? Or just as a token-tree?</p>
</blockquote>
<p>The compiler needs to identify where in the input there might be a macro invocation, which means it needs to do 'real' parsing at some point (we could push that back to when the user calls <code>expand</code> though, and just do token tree parsing here like you say). If we do eager parsing, I'm unsure as to whether we need grammar-context-aware constructors (<code>from_expr_tokens</code>, <code>from_item_tokens</code>, etc).</p>
<p>One alternative is to force <code>from_tokens</code> to only accept a single 'direct and complete' invocation (accepting <code>foo!()</code> but rejecting <code>let x = 5; foo!()</code>) and leaving it up to the caller to find and extract any macros in the input, but this causes issues if we ever want to support 'location aware' macro expansion (e.g. handling <code>super</code> when expanding <code>mod a { super::foo!{} }</code>).</p>



<a name="177203465"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/eager%20expansion/near/177203465" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edward Pierzchalski <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/eager.20expansion.html#177203465">(Oct 02 2019 at 23:48)</a>:</h4>
<blockquote>
<p>One goal I have is to try to revise the text so that we can illustrate the API by showing the <em>definition</em> of the declarative <code>expand!</code> in an appendix.</p>
</blockquote>
<p>That's a good idea, and I hoped to do so in an earlier draft, but I didn't feel like dealing with the details of input parsing and interpolation at the time. I might try and take a crack at it this weekend.</p>



<a name="177225643"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/eager%20expansion/near/177225643" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/eager.20expansion.html#177225643">(Oct 03 2019 at 08:55)</a>:</h4>
<blockquote>
<p>If we do eager parsing, I'm unsure as to whether we need grammar-context-aware constructors (<code>from_expr_tokens</code>, <code>from_item_tokens</code>, etc).</p>
</blockquote>
<p>Yes, this is exactly the problem I was anticipating.</p>



<a name="177225702"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/eager%20expansion/near/177225702" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/eager.20expansion.html#177225702">(Oct 03 2019 at 08:56)</a>:</h4>
<p>The good news is that a Builder interface like this lets you add new constructors later</p>



<a name="177225832"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/eager%20expansion/near/177225832" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/eager.20expansion.html#177225832">(Oct 03 2019 at 08:58)</a>:</h4>
<p>So I suspect the "right" answer will be delay the identification of macro invocations to the point where the user calls <code>expand</code>. (I'm trying to guess the value one would get from doing the identification of the invocations more upfront, and to be honest I'm drawing a blank, since its not like the ExpansionBuilder API allows one to <em>inspect</em> those identified invocations, right?)</p>



<a name="177294532"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/eager%20expansion/near/177294532" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edward Pierzchalski <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/eager.20expansion.html#177294532">(Oct 03 2019 at 22:26)</a>:</h4>
<p>Right, so maybe there's no harm in going all the way in the other direction and making <code>from_tokens</code> infallible, moving <em>all</em> the parsing logic and error reporting up to <code>expand</code>.</p>



<a name="177294721"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/eager%20expansion/near/177294721" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edward Pierzchalski <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/eager.20expansion.html#177294721">(Oct 03 2019 at 22:30)</a>:</h4>
<p>I suppose it's worth thinking a bit more about how the compiler will look for invocations in a token stream, see if we need those <code>_expr</code> variants for an MVP anyway</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>